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DESCRIPTION ^^ISCUIQ 
Enccding/Trammitting Apparatus and EnccKding/Transmitting Method 

Technical Field 

The present invention relates to an encoding/transmitting apparatus and an 
encoding/transmitting method. More particularly, the invention relates to an apparatus 
and method for transmitting coded data, which are fit for use in television-meeting systems, 
television-telephone systems, broadcasting systems and multi^nedia database systems in 
which moving^icture data is transmitted from a transmitting side to a receiving side 
through a transmission path and subjected to real-time playback (streaming). 

This application claims priority of Japanese Patent Application No. 2(X)2~359726, 
filed on December 11, 2(X)2, the entirety of which is incorporated by ref^ence herein. 

Background Art 

In recent years, video-data converting methods are widely used in distributing data 
between the broadcasting stations and the households. These mefliods accomplish 
transmission and storage of data at high efficiency, by utilizing the redundancy inherent to 
video data when the video data is treated as digital data. 

Among these video-data converting methods is an image-encoding method that is 
standardized by MPEQ4 (Moving Picture Expat Group). This image-encoding method 
is defined in ISO/DEC 14496 and is used in various applications, from those for 



professional use to those for consumer use. 

MPEG-4 is a system that compresses moving^icture data. The MPEG-4 
standards provide a system for achieving realtime playback (hereinafter called 
"streaming") of moving^icture data, particularly in television-meeting systems, 
television-telephone systems, broadcasting systems and multimedia database systems. If 
data is encoded and transmitted in accordance witti MPEG-4, the receiving-side system 
performs error correction and interleaving. Depending on the traffic condition in the 
transmission path, however, the packets are inevitably lost or the data inevitably has errors, 
To perform streaming on the network, it is, therefore, necessary to control the bit rate of the 
data to output and to control the amount of data passing through the transmission path. 

FIG 1 shows a conventional encoding/transmitting ^paratus 100. As FIG 1 
shows, the encoding/transmitting apparatus 100 comprises a video encoding unit 101, an 
audio encoding unit 102, an encoded-^ideo-data storage unit 103, an encoded-audio-data 
storage unit 104, and a multiplex output unit 106. The video encoding unit 101 encodes 
the input video data. The audio encoding unit 102 encodes the input audio data. The 
encoded-^ideo-data storage unit 103 stores the video data encoded. The 
encoded-audio-data storage unit 104 stores the audio data encoded. The multiplex output 
unit 106 multiplexes tiie video data and the audio data, both encoded, and outputs them. 
The encoding/transmitting apparatus 100 further comprises a network-state determining 
unit 105. The network-state detCTniining unit 105 receives information about packet loss, 
from the communication party. If the unit 105 determines that the load on the network is 
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heavy, it controls the video encoding unit 101 and audio encoding unit 102 to lower the 
encoding bit rate. 

Note that, in FIG 1, the broken lines indicate the flow of data, and the solid lines 
indicate the flow of control signals. 

Another method is available to control the encoding bit rate. This method is to 
lower tfie encoding bit rates of the video-data encoding unit and audio-data encoding unit, 
on the basis of the buffer-occupied storage areas in the data storage units. (See Japanese 
Patent Application Laid-Open Publication No. 1 1^1608.) 

The encoding bit rates may be controlled in accordance with a response from the 
communication party. However, no response can hardly received fix)m the 
communication party if congestion occurs on the network. In this case, the network-state 
determining unit cannot determine tiie state of the network. Consequently, the bit rates 
may not be appropriately controlled. Further, the bit rates cannot be controlled in 
accordance wifli the state of the network, by monitoring only the bufier-occupied storage 
areas. 

Disclosure of the Invention 

An object of this invention is to provide a novel encoding/transmitting apparatus 
and a novel encoding/transmitting method, each capable of solving the problems with the 
prior art described above. 

Another object of the invention is to provide an encoding/transmitting apparatus 
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and an encoding/transmitting method, which perform an appropriate rate control even if the 
load on the transmission path increases. 

An encoding/transmitting apparatus according to this invention conprises: an 
input means for inputting data; an encoding means for encoding the data input; a storage 
means for storing encoded data generated by the encoding means; a multiplexing means 
for multiplexing the encoded data stored in the storage means and transmitting tiie data 
multiplexed, to a predetermined receiving apparatus connected through a network; and a 
monitoring means for monitoring a state of the network. The multiplexing means 
controls a multiplexing rate in accordance with the state of the network, which the 
monitoring means has detected. 

An encoding/transmitting method according to the invention con:q)rises: a step of 
inputting data; a step of encoding the data input; a step of storing, in the storage means, 
encoded data generated in the step of encoding the data; and a step of multiplexing the 
encoded data stored in the storage means and transmitting the data multiplexed, to a 
predetermined receiving apparatus connected through a network. In the step of 
multiplexing the encoded data, a state of the network is input and a multiplexing rate is 
controlled in accordance with the state of the network, which has been input. 

The other objects of this invention and the advantages attained by this invention 
will be more apparent from the embodiments of the invention, which will be described 
with reference to flie accon^anying drawings. 
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Brief Description of Drawings 

FIG 1 is a block diagram showing a conventional encoding/transmitting 
apparatus; 

FIG 2 is a block diagram showing an encoding/transmitting apparatus according 
to this invention; 

FIG 3 is a diagram illustrating a state that encoded data occupies a small area in 
the encoded- video-data storage unit; 

FIG 4 is a diagram illustrating a state that encoded data occupies a large area in the 
encoded-video-data storage unit; 

FIGS. 5A to 5C are charts explaining the opraation of ttie encoding/transmitting 
apparatus; 

FIG 6 is a block diagram depicting another encoding/transmitting apparatus 
according to this invention; 

FIG 7 is a block diagram showing still another encoding/transmitting apparatus 
according to the present invention; 

FIGS. 8A to 8C are charts explaining the opaation of the encoding/transmitting 
apparatus shown in FIG 7; and 

FIG 9 is a block diagram depicting another encoding/transmitting apparatus 
according to the present invention. 

Best Mode for Carrying out the Invention 



5 



EnccKling/lrammitting apparatuses according to the present invention will be 
described, with reference to the accompanying drawings. Any encoding/transmitting 
apparatus according to this invention encodes streaming data and outputs the data thus 
encoded to external apparatuses. It decodes images and sounds sequentially input and 
outputs them to the external apparatuses. 

As shown in FIG 2, an encoding/transmitting apparatus 1 comprises a video 
encoding unit 11, an audio encoding unit 12, an encoded-A^ideo-data storage unit 13, an 
encoded-audio-<lata storage unit 14, a multiplex output unit 15, and an extemal-output 
request determining unit 16. The video encoding unit 11 encodes the input video data. 
The audio encoding unit 12 encodes the input audio data. The encoded-A^ideo-data 
storage unit 13 stores the video data encoded. The encoded-audio-data storage unit 14 
stores the audio data encoded. The multiplex output unit 15 multiplexes the video data 
and the audio data, both encoded, and outputs them. The external-output request 
determining unit 16 monitors the network and determines whether an output request has 
been made. 

Note that, in FIG 2, the broken lines indicate the flow of data, and the solid lines 
indicate the flow of control signals. 

\^deo data items, which are distributed in the form of a stream, are sequentially 
input to the video encoding unit 1 1 . The video encoding unit 1 1 encodes the input video 
data in accordance with encoding rules such as the mles of MPEG-4 (Moving Picture 
Expert Group). The video data encoded by the video encoding unit 11 is stored in the 
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encx)ded-^ideo-data storage unit 13. 

While encoding the video data, the video encoding unit 11 receives a 
multiplexing-completion signal from the multiplex output unit 15. The 
multiplexing-con^letion signal is a signal that the multiplex output unit 15 generates upon 
finishing the multiplexing. This signal contains information identifying the encoded data 
that has been multiplexed. The video encoding unit 11 releases the encoded video data 
from the encoded-video-data storage unit 13, on the basis of the information described in 
the multiplexing-completion information. The word "release" means a process of 
allowing other encoded data to be stored in the storage area that was occupied by any 
encoded data. 

The video encoding tmit 11 discriminates the encoded data read from the 
encoded-^^ideo-data storage unit 13 by the multiplex output unit 15, on the basis of the 
information described in the multiplexing-completion signal. 

The video encoding unit 11 manages the data now stored in the 
encoded-video-data storage unit 13, in accordance with the encoded data thus 
discriminated. The unit 11 then detemiines the area that the data occupies in the 
encoded-video-data storage unit 13. 

That is, the video encoding unit 11 calculates the area the encoded data occupies in 
the encoded-video-data storage unit 13, from the difference between the amount of 
encoded video data and the amount of the multiplexed, encoded video data. The video 
encoding unit 11 temporarily stops encoding data or keeps encoding data, in accordance 
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with how large or small an area that the encoded data occupies in the encoded-A^ideo-data 
storage unit 13. 

Audio data items are sequentially input to the audio encoding unit 12. The audio 
encoding unit 12 encodes the input audio data in accordance with encoding rules such as 
the MPEG mles. The audio data encoded by the audio encoding unit 12 is stored in the 
encoded-audio-data storage unit 14. The audio encoding unit 12 receives the 
multiplexing-con^letion signal from the multiplex output unit 15. In accordance with the 
multiplexing-completion signal, the audio encoding unit 12 releases the multiplexed, 
encoded audio data from the encoded-audio-<iata storage unit 14. 

The audio encoding unit 12 encodes the input audio data. It calculates the area 
the encoded data occupies in the encoded-audio-data storage unit 14 from the difference 
between the amount of encoded audio data and the amount of the multiplexed, encoded 
audio data. The audio encoding unit 12 temporarily stops encoding data or keeps 
encoding data, in accordance with how large or small an area that the encoded audio data 
occupies in the encoded-audio-data storage unit 14. 

The multiplex output unit 15 reads the data from the encoded-^ideo-data storage 
unit 13, and the data from the encoded-audio-data storage unit 14. The unit 15 
multiplexes the encoded video data and the encoded audio data. The multiplexed data is 
output to the network. Upon finishing multiplexing the encoded video data and the 
encoded audio data, the multiplex ou^ut unit 15 gen^ates a multiplexing-completion 
signal. 
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The multiplexing-completion signal is a signal that identifies the encoded video 
data that has been multiplexed. The video encoding unit 11 and the audio encoding unit 
12 release the encoded video data and encoded audio data, which have been multiplexed, 
from memories such as the encoded^deo-data storage unit 13 the encoded-audio-data 
storage unit 14, in accordance with the multiplexing-con5)letion signal. 

The external-output request determining unit 16 monitors the network. When 
the network assumes an imdesirable state, the unit 16 outputs a multiplexing stop 
command, causing the multiplex output unit 15 to stop the multiplexing. 

How the encoding/transmitting apparatus 1 that has the above-described 
configuration opiates to output video data to the network will be described, with reference 
to FIGS. 3 to 5. In the following description, how the apparatus 1 transmits video data 
will be explained first, and how it transmits audio data will be explained next. 

To transmit data, the video encoding unit 11, the multiplex output unit 15, and the 
multiplex-output request determining unit operate in parallel. 

The operation of the video encoding unit 11 will be described first. The video 
encoding unit 11 encodes the video data input to it, generating encoded video data. The 
video encoding unit 11 outputs the encoded video data to the encoded-A^ideo-data storage 
unit 13. The encoded-^deo-data storage unit 13 stores the input video data items, one 
after another. 

Every time the video encoding unit 11 encodes video data, it determines the area 
that tiie encoded data occupies in the encoded-video-data storage unit 13. Various 
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methods are available, which the unit 11 may perforai to determine this area. For 
example, the area is calculated from the difference between the amount of the encoded data 
and the amoimt of the multiplexed data. 

If the vacant area is large in the encoded-^deo-data storage unit 13 as is illustrated 
in FIG 3, the video encoding unit 11 encodes the input video data. The encoded video 
data is accumulated in the encoded-^ideo-data storage unit 13. 

If the vacant area is small in the encoded^ideo-data storage unit 13 as is shown in 
FIG 4, the video encoding unit 1 1 stops encoding the input video data. In this case, a part 
of the input video data is not encoded, and some frames of video data are not stored in the 
unit 13. 

The op^tion of the multiplex output unit 15 will be described. The multiplex 
output unit 15 operates as the video encoding unit 11 encodes the input video data. The 
unit 15 multiplexes the data stored in the encoded^ideo-data storage unit 13, outputting 
multiplexed data to the network. Upon finishing multiplexing the encoded data, the 
multiplex output unit 15 outputs a multiplexing-completion signal to the video encoding 
unit 11. When the video encoding unit 11 receives the multiplexing-completion signal, it 
releases the multiplexed, encoded video data from the encoded^ideo-data storage unit 13. 

As indicated above, the video encoding unit 11 encodes the input video data and 
the multiplex output unit 15 multiplexes the encoded video data and outputs the resultant 
multiplexed data to the network. The area that the encoded data occupies in the 
encoded-video-data storage unit 13 increases every time an item of encoded video data is 
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stored into the unit 13, and decreases every time an item of encoded video data is released 
from the unit 13. 

The operation of the external-output request determining unit 16 will be described. 
The determining unit 16 monitors the network as the video encoding unit 11 and the 
multiplex ou^ut unit 15 operate. When the network assumes an undesirable state, the 
unit 16 outputs a multiplexing stop command to the multiplex output unit 15. 

Upon receiving the multiplexing stop command, the multiplex ou^ut unit 15 stops 
multiplexing data, in accordance with tiiis command. The video encoding unit 11 keeps 
encoding the input video data even if the unit 15 stops multiplexing data. Hence, the 
encoded^ideo-data storage unit 13 accumulates the encoded video data. The video 
encoding unit 1 1 goes on encoding the input video data until the vacant storage area of the 
encoded-^ideo-<iata storage unit 13 becomes very smaU. The unit 11 stops encoding the 
input video data when the vacant storage area becomes insufficient 

The operation of the encoding/transmitting apparatus 1 will be desaibed in greatCT 
detail, with reference to FIG 5. FIG 5 is charts explaining flie operation of the 
encoding/transmitting apparatus 1. FIG 5 A represents the relation between the input 
video data, the encoded video data stored in the encoded^ideo-data storage unit 13 and the 
encoded video data output to the network. In FIG 5A, In(i) is the input video data, ES(i) 
is the encoded video data, Out(i) is the encoded video data output (hereinafter referred to as 
"output data"). The suffix (i) put to the three types of data indicates that these data items 
are related to one another. 
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In FIG 5A, the arrow directed bom the video data In(i) to the encoded data Es(i) 
shows that the video data Li(i) has been encoded into the data Es(i). Similarly, the arrow 
directed from the encoded data Es(i) to the ou^ut data Out(i) indicates that the encoded 
data Es(i) has been multiplexed, and the arrow directed from the output data Out(i) to the 
encoded data Es(i) indicates that a release request has been output for the area in which the 
encoded data Es(i) is stored. 

FIG 5B is a chart showing when the times at which the state of the 
encoding/transmitting apparatus 1 changes. In FIG 5B, point A indicates the time when a 
multiplexing stop command is output because the network has assumed an undesirable 
state. Point B indicates the time when the area that the encoded data occupies in the 
encoded-^ideo-data storage unit 13 exceeds a predetermined value. Point C indicates the 
time when a multiplexing start command is output because the network has restored a good 
state. Point D indicates the time when the encoded data is released and the area that the 
encoded data occupies in the encoded-A^ideo-data storage unit 13 decreases below the 
predetermined value. 

FIG 5C is a chart illustrating how the area that the encoded data occupies in the 
encoded^ideo-data storage unit 13 changes with time. In this figure, time is plotted on 
the axis of abscissa, and the area occupied by the encoded data in the encoded-^ideo-data 
storage unit 13 is plotted on the axis of ordinate. 

The video data In(i) input will be explained first. The encoding/transmitting 
apparatus 1 according to this embodiment is an apparatus that transmits stream data. This 
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is why video data items ln(0), In(l), In(n+4) are input to the encoding/transmitting 
apparatus 1, one after another at a regular pace. 

The video encxxling unit 11 encodes the input video data items ln(0), In(l), In(2), 
In(3), one by one, generating encoded video data items Es(0), Es(l), Es(2) and Es(3). 
The encoded video data items thus gen^ated are stored in the encoded-A^ideo-data storage 
unit 13. The multiplex output unit 15 multiplexes the encoded video data items Es(0), 
Es(l), Es(2) and Es(3), generating output data items C)ut(0), Out(l), Out(2) and Out(3), 
which are output to the network. 

Every time the multiplex output unit 15 multiplexes an encoded video data item, 
i.e., Es(0), Es(l), Es(2) or Es(3), the unit 15 generates a multiplexing-completion signal, 
which is output to the video encoding unit 11. In accordance with the 
multiplexing-conpletion signal, the video encoding unit 11 releases the multiplexed, 
encoded video data. 

During this operation, the vacant area for storing data, which is available in the 
encoded-A^ideo-data storage unit 13, decreases every time a video data item is encoded, and 
increases every time en encoded video data item is multiplexed, as is illustrated in FIG 5C. 
Since the encoding and the multiplexing are well balanced during the operation, the vacant 
storage area is stable. 

At time A, the network assumes an undesirable state. Upon detecting that the 
network has assumed the undesirable state, the determining unit 16 outputs a multiplexing 
stop command to the multiplex ou^ut unit 15. In response to the multiplexing stop 
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command, the multiplex output unit 15 stops multiplexing data. 

Even after flie multiplexing is stopped, video data items In(5), In(6), . . In (n-1) 
are sequentially input to the video encoding unit 1 1 . The video encoding unit 1 1 encodes 
the video data items In(5), In(6), . . In (n-1) input to it, genorating encoded video data 
items Es(5), ES(6), . . Es(n-1), even after the multiplexing is stopped. 

The video encoding unit 11 calculates the area that tiie encoded data occupies in 
the encoded-^ideo-data storage unit 13, before the encoded video data is stored in the unit 
13. The unit 11 keeps encoding the input video data until the area that the encoded data 
occupies in the encoded-^ideo-data storage unit 13 reaches the predetermined value. At 
this time, the multiplex output unit 15 is not operating. Hence, the area occupied by the 
encoded data only increases in the encoded^ideo-data storage unit 13. 

The area that the encoded data occupies in tiie encoded-^deo-data storage unit 13 
exceeds the predetermined value at time B. When the area occupied by the encoded data 
in the encoded-^deo-data storage unit 13 exceeds the predetermined value, the video 
encoding unit 1 1 stops encoding data. 

The "predetermined value" is a storage capacity that is required to encode an 
image. This storage capacity is, for example, one for storing an I picture obtained by 
encoding a con5)licated image, or for storing video data encoded at low bit rate. 

To change the encoding bit rate, the video encoding unit 11 carries out a rate 
control in accordance with the vacant area available in the encoded-video-data storage unit 
13. As the vacant area increases, the unit 11 can raise the bit rate. This rate control 
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makes it possible to distribute images, each missing no frames, though the image quality 
may decrease. 

Even if the video encoding unit 11 stops encoding video data, it receives video 
data items In(n), In(n+1) and In(n+2) input to it The video encoding unit 11 does not 
encode these input data item, nonetheless. The video data items In(n), In(n+1) and 
In(n+2) input to flie unit 1 1 are, flierefore, discarded. 

When the network restores a good state at time C, the determining unit 16 outputs 
a multiplexing start command to the multiplex output unit 15. Upon receiving the 
multiplexing start command, the multiplex output unit 15 reads, from the 
encoded-^ideo-data storage unit 13, the video data item Es(4) that has been first encoded, 
and multiplexes this video data item, generating output data Out(4). The output data 
Out(4) is smoothly transmitted to the network, because the network assumes a good state at 
this time. 

Upon multiplexing the encoded video data item Es(4), the multiplex output unit 15 
outputs a multiplexing-completion signal to the video encoding unit 11. In accordance 
with the multiplexing-<X)mpletion signal, the video encoding unit 11 releases the encoded 
video data item Es(4) from the encoded^ideo-<lata storage unit 13. 

When the encoded video data is released, the area that the encoded video data 
occupies decreases in the encoded-video-data storage unit 13. When the area occupied by 
the encoded video data decreases, the video encoding unit 11 starts performing encoding 
again, encoding the video data In(n+4) input next, thus generating an encoded video data 
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itemEs(n+4). 

The multiplex oulput unit 15 multiplexes the encoded video data items Es(4), 
Es(5), , . . , Es(n-1), in the order these data items have been generated. The encoded video 
data items, thus multiplexed, are oulput to the network. Every time an encoded video 
data item is multiplexed, the area occupied by encoded video data decreases in the 
encoded-A^ideo-data storage unit 13. Every time a video data item is encoded, the area 
increases. The encoding and the multiplexing are well balanced during tiie operation. 
Hence, the vacant storage area is stable. 

In the encoding/transmitting apparatus 1 according to this invention, the 
external-output request determining unit 16 monitors the state of the network as described 
above. On the basis of what the unit 16 has determined, it is determined whether the 
process of transmitting data should be stopped or started again. 

In the encoding/transmitting apparatus 1 according to this invention, the 
external-output request determining unit 16 that monitors the state of the network is not 
provided on the network. Even if no response comes from the communication party, the 
state of the network can be confirmed from the output of the external-output request 
determining unit the unit 16. 

How video data is processed has been described above. Audio data is processed 
in almost the same way. The above description holds true of audio data, too, only if the 
video encoding unit 11 and the encoded^ideo-data storage unit 13 are taken for the audio 
encoding unit 12 and the encoded-audio-data storage unit 14, respectively. Thus, how 
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audio data is processed will not be explained. 

An encoding/transmitting apparatus 2, which is another embodiment of this 
invention, will be described. This apparatus 2 con^wises a filter 27 that is connected to 
the input of the audio encoding unit 22 as is illustrated in FIG 6. 

The filter 27 is a filter that causes sound to fade in and fade out. 

The filter 27 is controlled by, for example, the audio encoding unit 22. How the 
audio encoding unit 22 controls the filter 27 will be described. As indicated above, every 
time the audio encoding unit 22 starts encoding audio data, it checks the area that encoded 
audio data occupies in the encoded-audio-data storage unit 24. If the area occupied by the 
encoded audio data is so large that audio data may no longer be encoded, the unit 22 
controls the filter 27, causing the same to perform fadingout of the audio data to be 
encoded next If the area occupied by the encoded audio data is so smaU that audio data 
may be encoded, the unit 22 controls the fiilter 27, achieving fading^ of the audio data to 
be encoded next. 

In FIG 6, too, the broken lines indicate the flow of data, and the solid lines indicate 
the flow of control signals. 

The encoding/transmitting apparatus shown in FIG 6 has the filter 27 in order not 
to induce strangeness, which may otherwise result fi'om firames missing in audio data. 
When the unit 22 stops encoding audio data, some frames of the audio data are lost. As a 
consequence, the sound becomes discontinuous. The filter 27 causes sound to fade in or 
out. Thus, each acoustic part smoothly reaches an anacoustic part, and each anacoustic 
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part smoothly reaches an acoustic part. 

An encoding/transmitting apparatus 3, which is still another embodiment of tfiis 
invention, will be described. This apparatus 3, which is shown in FIG 7, con^rises a 
multiplex ou^ut unit 35 that has a shaping function. 

In FIG 7, too, the broken lines indicate the flow of data, and the solid lines indicate 
the flow of control signals. 

Video data changes in amount, in depending on the complexity, motion and type 
of the image it represents and the type of encoded data. The shaping function is a 
function of outputdng encoded data from the encoded-video-data storage unit to the 
network at a predetemiined speed. 

Assume that video data is encoded in accordance with the MPEG encoding mles. 
Then, the image is converted to an I picture, a B picture, or a P picture. The I picture is an 
image that is basic to the B picture and the P picture. The data representing an I picture is 
greater than the data representing a B picture or a P picture. The P picture is an image that 
is basic to the B picture. The data representing a P picture is greater than the data 
representing a B picture. Hence, the amount of encoded data changes, in accordance with 
the type of the picture it represents. The shaping function suppresses the change in the 
amoimt of data to be transmitted, which depends on the change in the amount of encoded 
data. 

FIG 8 is charts explaining how the encoding/transmitting apparatus 3 shown in 
FIG 7 transmit data. In FIG 8A, In(i) denotes the video data input, Es(i) indicates the 
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video data encoded, and C)ut(i) denotes the video data to be output to the network, as in FIG 
5A. 

FIG 8B is a chart shov^g when the times at which the state of the 
encoding/transmitting apparatus 3 changes. In FIG 8B, point O indicates the time when 
the area that the encoded data occupies in the encoded^ideo-<iata storage unit 33 exceeds a 
predetermined value. Pont P indicates the time when the encoded data is released due to 
multiplexing and the area occupied by the encoded data decreases below a predetmnined 
value. 

FIG 8C is a chart illustrating how the area occupied by the encoded data in the 
encoded-video-data storage unit 33 changes with time. In this figure, time is plotted on 
the axis of abscissa, and the area occupied by the encoded data in the encoded-video-data 
storage unit 33 is plotted on the axis of ordinate. 

How the encoding/transmitting apparatus 3 transmits data will be described, with 
reference to FIG 8. The encoding/transmitting apparatus 3 is an apparatus that encodes 
streaming data and transmits the data thus encoded to external apparatuses. This is why 
video data items ln(0), In(l), In(2), ... are input to the encoding/transmitting apparatus 3, 
one aftCT another. 

The video encoding unit 31 encodes the input video data items ln(0), In(l), In(2), 
. . ., one by one. The video data item ln(0) input first is converted to encoded video data 
item Es(0). The encoded video data item Es(0) represents an I picture and is large in 
amount. 
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The video encoding unit 31 then converts the video data item In(l) input next, to 
an encoded video data item Es(l). This encoded video data item Es(l) represents a P 
picture or a B picture. It is smaller in amount than the encoded video data item Es(0) first 
generated. 

The multiplex output unit 35 stores data representing the maximum transmission 
amount in which data can be transmitted. The unit 35 reads encoded video data from the 
encoded-^ideo^ta storage unit 33, so that the amount of data read may not exceed the 
maximum transmission amount. The multiplex output unit 35 divides the encoded video 
data read from the unit 33, into blocks of an appropriate size and multiplexes the data 
blocks. The data blocks are output to the network. 

As described above, the encoded video data items Es(0), Es(l), Es(2), . . . differ in 
amount. The multiplex output unit 35 divides the encoded video data into blocks of an 
appropriate size. More precisely, the unit 35 divides the video data item Es(0) into four 
blocks, and Es(l) and Es(2) each into two blocks and multiplexes these blocks, generating 
multiplex data. The multiplexed data is output to the network. 

Thus, the data item Es(0) is divided into a plurality of blocks, which are 
sequentially transmitted. The timing of encoding the video data and the timing of 
transmitting the encoded video data therefore differ from each other. 

In the instance of FIG 8, Es(l), Es(2) and Es(3) have been encoded at the time 
Es(0) is transmitted in its entirety. Es(l), Es(2) and Es(3) are stored in the 
encoded^ideo-data storage unit 33. The encoded data is transmitted at a constant rate. 
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Hence, if the encoded video data is large in amount, the following encoded video data will 
be accumulated in the encoded-A^ideo-data storage unit 33 before the video data is 
transmitted. At time O, the area occupied by the encoded data in the encoded-video-<iata 
storage unit 33 inevitably exceeds the predetermined value. 

When the area occupied by the encoded data exceeds the predetermined value, the 
video encoding unit 31 stops encoding video data. The video data items Li(4) and Li(5), 
both input after time O, are therefore discarded, without being encoded. 

The multiplex output unit 35 keeps multiplexing the encoded video data and 
output it to the network, even while the video encoding unit 31 is encoding no data. The 
multiplex output unit 31 divides the encoded video data into blocks and outputs the blocks, 
one by one, to the network. Every time the multiplex output unit 35 outputs a block, it 
outputs a multiplexing-completion signal to the video encoding unit 31 . 

When the video encoding unit 31 receives the multiplexing-conpletion signal, it 
releases the encoded video data from the encoded^ideo-data storage unit. Every time flie 
video encoding unit 31 receives a video data item, it checks the area that encoded audio 
data occupies. In the case shown in FIG 8, the area occupied by the encoded data 
becomes smaller than the predetermined value at time P. Then, the encoding is started 
again. 

The video encoding unit 3 1 encodes the video data items In(7), In(8), . . ., one after 
another, generating encoded video data items Es(7), Es(8), .... The encoded video data 
items Es(7), Es(8), ... are stored in the encoded-video-data storage unit 33. The multiplex 
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output unit 35 divides the encxxled video data items Es(7), Es(8), each into blocks. 
The blocks are output to the network. Every time the multiplex output imit 35 
multiplexes a block, it generates a multiplexing-completion signal to the video encoding 
unit 31. If the network assumes an undesirable state, the extemal-ou^ut request 
determining unit 36 outputs a multiplexing stop command to the multiplex output unit 35. 
The unit 36 therefore causes the multiplex output unit 35 to stop multiplexing data. 

In the encoding/transmitting apparatus 3 that has a shaping fiinction, flie encoding 
process is thus controlled. With the encoding/transmitting apparatus 3 that has a shaping 
function, it is possible to control the amount in which the data is transmitted. In addition, 
it is possible to control the frame rate if the network assumes an undesirable state. 

How video data is processed in the apparatus shown in FIG 7 has been described 
above. Audio data is processed in almost the same way. To prevent the amount of data 
to transmit from exceeding the maximum transmission amount, the multiplex output unit 
may perform a shaping process and two controls. In the shaping process, it divides audio 
data into blocks. In the first control, it stops multiplexing data, when it receives a 
multiplexing stop command from the external-output request determining unit. In the 
second control, it causes the audio encoding unit to stop encoding audio data when the 
storage capacity of the encoded-audio-data storage unit becomes decreases too much. 

An encoding/transmitting apparatus 4, which is a fiirther embodiment of this 
invention, will be described. As shown in FIG 9, this apparatus 4 outputs stream data that 
contains a plurality of programs. The encoding/transmitting apparatus 4 comprises as 



22 



many video encoding units 41 as the programs, as many audio encoding units 42 as the 
programs, as many encoded-A^ideo-data storage units 43 as the programs, and as many 
encoded-audio-data storage units 44 as the programs. 

In FIG 9, too, the broken lines indicate the flow of data, and the solid lines indicate 
the flow of control signals. 

The video data and audio data of a certain program are output to ttie video 
encoding unit 41 and audio encoding unit 42 that correspond to the program. The 
multiplex output unit 45 is connected to aU video encoding units 41 and all audio encoding 
units 42. The multiplex output unit 45 multiplexes the encoded video data items read 
from the video encoding units 41 and the encoded audio data items read from the audio 
encoding units 42, generating one stream data item. 

In the encoding/transmitting apparatus 4 shown in FIG 9, video encoding units 41, 
audio encoding units 42, encoded-^ideo-data storage units 43 and encoded-audio-data 
storage units 44 operate almost in the same way as their counterparts provided in the 
encoding/transmitting apparatus 4 according to the first embodiment However, the 
multiplex output unit 45 pCTforms an operation that differs from the operation that its 
counterpart does in the encoding/transmitting apparatus 4 according to the first 
embodiment 

To be more specific, when the multiplex output unit 45 multiplexes the data items 
read from the encoded-^ddeo-data storage units 43 and encoded-audio-data storage unit 44, 
it records as to which encoded video data item has been read from which 
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encoded-video-data storage unit 43, and which encoded audio data item has been read 
from which encoded-audio-data storage unit 44. The multiplex output unit 45 outputs a 
multiplexing-completion signal to the video encoding unit 41 and audio encoding unit 42 
that correspond to the program, the data about which data has been read. 

The multiplex output unit 45 may have a shaping function, as in the 
encoding/transmitting apparatus 4 according to the second embodiment. Thus, the unit 
45 may divide the data stored in each encoded-video-data storage unit 43 and each 
encoded-audio-data storage unit 44 into blocks, and may ou^ut data in an amount not 
exceeding a predetermined amount. 

In the encoding/transmitting apparatus 4 shown in FIG 9, the multiplexed output 
contains data showing which data about which program has been multiplexed, a 
multiplexing-completion signal is output to the video encoding unit 41 and audio encoding 
imit 42 that correspond to the program. Thus, this invention can be applied to the 
encoding/transmitting apparatus 4, too, which distributes stream data containing a plurality 
of programs. 

As has been described, encoding/transmitting apparatuses according to the present 
invention have an external-output request determining unit that determines the state of the 
network. Hence, the apparatuses can detect the state of the network even if no responses 
come from the communication party. The apparatus can therefore accomplish an 
appropriate rate control. 

This invention can be applied to encoding/transmitting apparatuses that have 
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shaping fiinction or encoding/transmitting apparatuses that have a plurality of programs, 
too. 

The present invention is not limited to the embodiments described above. 
Various changes and modifications that include the gist of the invention should be 
considered to be fall within the scope and spirit of the invention. For example, each 
encoding/transmitting apparatus described above is configured to output stream data. The 
output data is not limited to stream data, nevertheless. Any encoding/transmitting 
apparatus that outputs data composed of images temporarily stored in an encoded data 
storage unit should be considered to fall within the scope of this invention. 

The multiplex ou^ut unit is designed to stop multiplexing data when the network 
assumes an imdesirable state. Instead, the multiplexing bit rate may be controlled, either 
raised or lowered, in accordance with the state of the network. 

In the embodiments described above, the video encoding unit and the audio 
encoding unit stop encoding data when the area occupied by encoded data increases too 
much. Nonetheless, the video encoding luiit and the audio encoding unit may be 
controlled to raise or lower the encoding bit rate. 

Industrial Applicability 

In the present invention, the monitoring means for monitoring the state of the 
network is incorporated, as described above, in an encoding/transmitting apparatus. Data 
transmission can therefore be controlled even if no data showing the receipt of data comes 
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from the communication party. Further, the data transmission can be controlled in real 
time, without waiting for the data coming from the communication party and indicating the 
receipt of data. Moreover, the load on the network can be reduced because the data 
transmission can be controlled, without receiving data via the network. In addition, ttie 
encoding means receives transmission signals directly from the multiplexing means and 
can therefore determine the state of the storage means. 
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